System and method for determining the three-dimensional location and orienation of identification markers

ABSTRACT

A three-dimensional position and orientation tracking system comprises one or more pattern tags, each comprising a plurality of contrasting portions, a tracker for obtaining image information about the pattern tags, a database with geometric information describing patterns on pattern tags; and a controller for receiving and processing the image information from the tracker, accessing the database to retrieve geometric information, and comparing the image information with the geometric information. The contrasting portions are arranged in a rotationally asymmetric pattern and at least one of the contrasting portions on a pattern tag comprising a perimeter with a polygonal shape. The pattern tags may be borne on tracking markers that have a three-dimensional shaped surface. The tracking system can be implemented in a surgical monitoring system in which the pattern tags are attached to tracking markers or are themselves tracking markers. A method associated with the system employs the rotationally asymmetric patterns on the tags to determine the three-dimensional locations and orientations of items bearing the tags using non-stereo image information.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 14/562,691, filed Dec. 6, 2014, now U.S. patentSer. No. ______, which is a continuation-in-part of PCT InternationalApplication Serial Number PCT/EP2013/073401, filed Nov. 8, 2013, and acontinuation-in part of U.S. patent application Ser. No. 13/713,165,filed Dec. 13, 2012, now U.S. Pat. No. 8,908,918, both of which claimpriority under 35 U.S.C. §119(e) of U.S. Patent Provisional ApplicationSer. No. 61/724,024, filed Nov. 8, 2012, the disclosures of which areincorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to location monitoring hardware and softwaresystems. More specifically, the field of the invention is that ofsurgical equipment and software for monitoring surgical conditions.

2. Description of the Related Art

Visual and other sensory systems are known, with such systems beingcapable of both observing and monitoring surgical procedures. With suchobservation and monitoring systems, computer aided surgeries are nowpossible, and in fact are being routinely performed. In such procedures,the computer software interacts with both clinical images of the patientand observed surgical images from the current surgical procedure toprovide guidance to the physician in conducting the surgery. Forexample, in one known system a carrier assembly bears at least onefiducial marker onto an attachment element in a precisely repeatableposition with respect to a patient's jaw bone, employing the carrierassembly for providing registration between the fiducial marker and thepatient's jaw bone and implanting the tooth implant by employing atracking system which uses the registration to guide a drillingassembly. With this relatively new computer implemented technology,further improvements may further advance the effectiveness of surgicalprocedures.

SUMMARY OF THE INVENTION

The present invention is a surgical hardware and software monitoringsystem and method which allows for surgical planning while the patientis available for surgery, for example while the patient is beingprepared for surgery so that the system may model the surgical site. Inone embodiment, the model may be used to track contemplated surgicalprocedures and warn the physician regarding possible boundary violationsthat would indicate an inappropriate location in a surgical procedure.In another embodiment, the hardware may track the movement ofinstruments during the procedure and in reference to the model toenhance observation of the procedure. In this way, physicians areprovided an additional tool to improve surgical planning andperformance.

The system uses a particularly configured fiducial reference, to orientthe monitoring system with regard to the critical area. The fiducialreference is attached to a location near the intended surgical area. Forexample, in the example of a dental surgery, a splint may be used tosecurely locate the fiducial reference near the surgical area. Thefiducial reference may then be used as a point of reference, or afiducial, for the further image processing of the surgical site. Thefiducial reference may be identified relative to other portions of thesurgical area by having a recognizable fiducial marker apparent in thescan.

The system of embodiments of the invention involves automaticallycomputing the three-dimensional location of the patient by means of atracking device that may be a tracking marker. The tracking marker maybe attached in fixed spatial relation either directly to the fiducialreference, or attached to the fiducial reference via a tracking polethat itself may have a distinct three-dimensional shape. In the dentalsurgery example, a tracking pole is mechanically connected to the baseof the fiducial reference that is in turn fixed in the patient's mouth.Each tracking pole device has a particular observation pattern, locatedeither on itself or on a suitable tracking marker, and a particulargeometrical connection to the base, which the computer softwarerecognizes as corresponding to a particular geometry for subsequentlocation calculations. Although individual tracking pole devices havedistinct configurations, they may all share the same connection base andthus may be used with any fiducial reference. The particular trackinginformation calculations are dictated by the particular tracking poleused, and actual patient location is calculated accordingly. Thus,tracking pole devices may be interchanged and calculation of thelocation remains the same. This provides, in the case of dental surgery,automatic recognition of the patient head location in space.Alternatively, a sensor device, or a tracker, may be in a known positionrelative to the fiducial key and its tracking pole, so that the currentdata image may be mapped to the scan image items.

The fiducial reference and each tracking pole or associated trackingmarker may have a pattern made of radio opaque material so that whenimaging information is scanned by the software, the particular items arerecognized. Typically, each instrument used in the procedure has aunique pattern on its associated tracking marker so that the trackerinformation identifies the instrument. The software creates a model ofthe surgical site, in one embodiment a coordinate system, according tothe location and orientation of the patterns on the fiducial referenceand/or tracking pole(s) or their attached tracking markers. By way ofexample, in the embodiment where the fiducial reference has anassociated pre-assigned pattern, analysis software interpreting imageinformation from the tracker may recognize the pattern and may selectthe site of the base of the fiducial to be at the location where thefiducial reference is attached to a splint. If the fiducial key does nothave an associated pattern, a fiducial site is designated. In the dentalexample this can be at a particular spatial relation to the tooth, and asplint location can be automatically designed for placement of thefiducial reference.

In a first aspect of the invention there is provided a surgicalmonitoring system comprising a fiducial reference configured forremovably attaching to a location proximate a surgical site, for havinga three-dimensional location and orientation determinable based on scandata of the surgical site, and for having the three-dimensional locationand orientation determinable based on image information about thesurgical site; a tracker arranged for obtaining the image information;and a controller configured for spatially relating the image informationto the scan data and for determining the three-dimensional location andorientation of the fiducial reference. In one embodiment of theinvention the fiducial reference may be rigidly and removably attachableto a part of the surgical site. In such an embodiment the fiducialreference may be repeatably attachable in the same three-dimensionalorientation to the same location on the particular part of the surgicalsite.

The fiducial reference is at least one of marked and shaped for havingat least one of its location and its orientation determined from thescan data and to allow it to be uniquely identified from the scan data.The surgical monitoring system further comprises a first tracking markerin fixed three-dimensional spatial relationship with the fiducialreference, wherein the first tracking marker is configured for having atleast one of its location and its orientation determined by thecontroller based on the image information and the scan data. The firsttracking marker may be configured to be removably and rigidly connectedto the fiducial reference by a first tracking pole. The first trackingpole can have a three-dimensional structure uniquely identifiable by thecontroller from the image information. The three-dimensional structureof the first tracking pole allows its three-dimensional orientation ofthe first tracking pole to be determined by the controller from theimage information.

The first tracking pole and fiducial reference may be configured toallow the first tracking pole to connect to a single unique location onthe fiducial reference in a first single unique three-dimensionalorientation. The fiducial reference may be configured for the attachmentin a single second unique three-dimensional orientation of at least asecond tracking pole attached to a second tracking marker. The firsttracking marker may have a three-dimensional shape that is uniquelyidentifiable by the controller from the image information. The firsttracking marker can have a three-dimensional shape that allows itsthree-dimensional orientation to be determined by the controller fromthe image information. The first tracking marker may have a marking thatis uniquely identifiable by the controller and the marking may beconfigured for allowing at least one of its location and its orientationto be determined by the controller based on the image information andthe scan data.

The surgical monitoring system may comprise further tracking markersattached to implements proximate the surgery site and the controller maybe configured for determining locations and orientations of theimplements based on the image information and information about thefurther tracking markers.

In another aspect of the invention there is provided a method forrelating in real time the three-dimensional location and orientation ofa surgical site on a patient to the location and orientation of thesurgical site in a scan of the surgical site, the method comprisingremovably attaching a fiducial reference to a fiducial location on thepatient proximate the surgical site; performing the scan with thefiducial reference attached to the fiducial location to obtain scandata; determining the three-dimensional location and orientation of thefiducial reference from the scan data; obtaining real time imageinformation of the surgical site; determining in real time thethree-dimensional location and orientation of the fiducial referencefrom the image information; deriving a spatial transformation matrix forexpressing in real time the three-dimensional location and orientationof the fiducial reference as determined from the image information interms of the three-dimensional location and orientation of the fiducialreference as determined from the scan data.

The obtaining of real time image information of the surgical site maycomprise rigidly and removably attaching to the fiducial reference afirst tracking marker in a fixed three-dimensional spatial relationshipwith the fiducial reference. The first tracking marker may be configuredfor having its location and its orientation determined based on theimage information. The attaching of the first tracking marker to thefiducial reference may comprise rigidly and removably attaching thefirst tracking marker to the fiducial reference by means of a trackingpole. The obtaining of the real time image information of the surgicalsite may comprise rigidly and removably attaching to the fiducialreference a tracking pole in a fixed three-dimensional spatialrelationship with the fiducial reference, and the tracking pole may havea distinctly identifiable three-dimensional shape that allows itslocation and orientation to be uniquely determined from the imageinformation.

In yet a further aspect of the invention there is provided a method forreal time monitoring the position of an object in relation to a surgicalsite of a patient, the method comprising removably attaching a fiducialreference to a fiducial location on the patient proximate the surgicalsite; performing a scan with the fiducial reference attached to thefiducial location to obtain scan data; determining the three-dimensionallocation and orientation of the fiducial reference from the scan data;obtaining real time image information of the surgical site; determiningin real time the three-dimensional location and orientation of thefiducial reference from the image information; deriving a spatialtransformation matrix for expressing in real time the three-dimensionallocation and orientation of the fiducial reference as determined fromthe image information in terms of the three-dimensional location andorientation of the fiducial reference as determined from the scan data;determining in real time the three-dimensional location and orientationof the object from the image information; and relating thethree-dimensional location and orientation of the object to thethree-dimensional location and orientation of the fiducial reference asdetermined from the image information. The determining in real time ofthe three-dimensional location and orientation of the object from theimage information may comprise rigidly attaching a tracking marker tothe object.

In one alternative embodiment, the tracker itself is attached to thefiducial reference so that the location of an object having a marker maybe observed from a known position.

In another aspect there is presented a three-dimensional position andorientation tracking system comprising at least one pattern tagcomprising a plurality of contrasting portions, a tracker configured forobtaining image information about the at least one pattern tag; adatabase comprising geometric information describing a pattern on the atleast one pattern tag; and a controller configured for receiving andprocessing the image information from the tracker; accessing thedatabase to retrieve geometric information; and comparing the imageinformation with the geometric information; characterized in that theplurality of contrasting portions are arranged in a rotationallyasymmetric pattern. The rotationally asymmetric pattern may be anidentifiably unique pattern. The at least one of the plurality ofcontrasting portions may have a perimeter comprising a mathematicallydescribable curved section. The perimeter of the at least onecontrasting portion may comprise a conic section, including an ellipseor a circle. The at least one pattern tag may be flexible. The at leastone pattern tag may be substantially planar. The at least one patterntag may be a tracking marker. The tracking marker may have a surfacethat is a segment of a three-dimensional surface and thethree-dimensional surface may be cylindrical or ellipsoid. The ellipsoidsurface may be a spherical surface. In some embodiments, the pluralityof contrasting portions may comprise a plurality of non-rectangularlyconfined contrasting portions arranged in a rotationally asymmetricpattern. In other embodiments the plurality of contrasting portions maycomprise a plurality of unconfined contrasting portions arranged in arotationally asymmetric pattern. In yet other embodiments, the pluralityof contrasting portions may have a predetermined random distribution.The tracker may be a non-stereo tracker.

In another embodiment the three-dimensional position and orientationtracking system comprises at least two pattern tags, a first of the atleast two pattern tags comprising a first plurality of contrastingportions and a second of the at least two pattern tags comprising atleast one contrasting portion, a tracker configured for obtaining imageinformation about the at least two pattern tags; a database comprisinggeometric information describing patterns on the at least two patterntags; and a controller configured for receiving and processing the imageinformation from the tracker; accessing the database to retrievegeometric information; and comparing the image information with thegeometric information; characterized in that at least one of the firstand second pattern tags has one or more contrasting portions arranged ina rotationally symmetric pattern; the contrasting portions of the firstand second pattern tags together constitute a rotationally asymmetricpattern. The rotationally asymmetric pattern may be an identifiablyunique pattern. The at least one contrasting portion of each of the atleast two pattern tags may have a perimeter comprising a mathematicallydescribable curved section. The perimeter of the at least onecontrasting portion may comprise a conic section, including an ellipseor a circle. The pattern tags may be flexible and may be substantiallyplanar. The at least two pattern tags together may constitute a trackingmarker. The pattern tags may be afixed to tracking markers and thetracking markers may have a shape that comprises at least a portionhaving a three-dimensional surface and the three-dimensional surface maybe cylindrical or ellipsoid. The ellipsoid surface may be a sphericalsurface.

In yet another aspect a method is provided for tracking an item bearingat least one pattern tag, the pattern tag bearing a rotationallyasymmetric pattern having a plurality of contrasting portions, at leastone of the plurality of contrasting portions having a perimetercomprising a mathematically describable curved section, the methodcomprising: obtaining from a tracker image information about the atleast one pattern tag; obtaining from a database geometric informationabout the at least one pattern tag, the geometric information comprisinga mathematical description of at least a section of the perimeter of theat least one contrasting portion of the at least one pattern tag;determining within the image information a three-dimensional location ofat least one pattern reference point of the at least one pattern tagbased on the geometric information, and determining within the imageinformation a three-dimensional rotational orientation of the at leastone pattern tag based on the geometric information. The plurality ofcontrasting portions may be a plurality of unconfined contrastingportions or non-rectangularly confined portions. In some embodiments ofthe method, the plurality of contrasting portions may have apredetermined random distribution; the geometric information may includepredetermined distribution information about the random distribution;and the determining the three-dimensional location of the at least onepattern reference point may therefore comprise comparing the imageinformation with the distribution information. The obtaining from atracker image information may be obtaining from the tracker non-stereoimage information.

In yet a further aspect there is provided a surgical monitoring systemcomprising a tracker for obtaining image information of a surgical site;a controller configured to spatially relate image information topreviously obtained scan data; a fiducial reference configured forremovably attaching to a location proximate the surgical site; atracking marker in fixed three-dimensional spatial relationship with thefiducial reference and observable by the tracker, the tracking markercomprising a plurality of contrasting portions arranged in arotationally asymmetric pattern; and controller software configured toallow the controller to determine the three-dimensional location andorientation of the fiducial reference based on the rotationallyasymmetric pattern. The rotationally asymmetric pattern may be anidentifiably unique pattern. The at least one of the contrastingportions may have a perimeter comprising a mathematically describablecurved section and the controller software may be configured to allowthe controller to determine at least one of the three-dimensionallocation and the orientation of the fiducial reference based on themathematically describable curved section. The perimeter of the at leastone contrasting portion may comprise a conic section, including anellipse or a circle. The tracking marker may have a shape that comprisesat least a portion having a three-dimensional surface and thethree-dimensional surface may be cylindrical or ellipsoid. The ellipsoidsurface may be a spherical surface. In some embodiments of the surgicalmonitoring system, the plurality of contrasting portions may comprise aplurality of non-rectangularly confined contrasting portions arranged ina rotationally asymmetric pattern. In other embodiments the plurality ofcontrasting portions may comprise a plurality of unconfined contrastingportions arranged in a rotationally asymmetric pattern. In yet otherembodiments, the plurality of contrasting portions may have apredetermined random distribution. The tracker may be a non-stereotracker.

In a further aspect a method is provided for tracking an item bearing apattern tag having at least one unique rotationally asymmetric pattern,the method comprising: obtaining image information from a tracker aboutthe at least one pattern tag; identifying the at least one pattern tagwithin the image information on the basis of its unique pattern;obtaining from a database geometric information about the at least onepattern tag, the geometric information comprising a mathematicaldescription of at least a section of the perimeter of at least onecontrasting portion of the at least one pattern tag; determining fromthe image information the location of at least one pattern referencepoint of the at least one pattern tag based on the geometricinformation; and determining from the image information thethree-dimensional rotational orientation of the at least one pattern tagbased on the geometric information and the at least one patternreference point. The obtaining image information from the tracker may beobtaining non-stereo image information from the tracker.

In another embodiment, a method is provided for tracking the locationand rotational orientation of an item comprising: attaching to the itema tracking marker bearing a rotationally asymmetric pattern ofcontrasting portions, at least a section of the perimeter of at leastone contrasting portion of the at least one pattern tag having amathematical description; obtaining image information from a trackerabout the at least one pattern tag; obtaining geometric information froma database about the at least one pattern tag, the geometric informationcomprising a mathematical description of the at least a section of theperimeter of at the least one contrasting portion of the at least onepattern tag; determining from the image information thethree-dimensional location of at least one pattern reference point ofthe at least one pattern tag based on the geometric information; anddetermining within the image information the three-dimensionalrotational orientation of the at least one pattern tag based on thegeometric information. The obtaining image information from a trackermay be obtaining non-stereo image information from the tracker. Theplurality of contrasting portions may have a predetermined randomdistribution and the geometric information may include predetermineddistribution information about the random distribution. In thisembodiment of the method the determining the three-dimensional locationof the at least one pattern reference point may comprise comparing theimage information with the distribution information.

In some embodiments, the perimeters of the contrasting portions may bemathematically describable polygonal shapes instead of havingmathematically describable curved sections.

BRIEF DESCRIPTION OF THE DRAWINGS

The above mentioned and other features and objects of this invention,and the manner of attaining them, will become more apparent and theinvention itself will be better understood by reference to the followingdescription of an embodiment of the invention taken in conjunction withthe accompanying drawings, wherein:

FIG. 1 is a schematic diagrammatic view of a network system in whichembodiments of the present invention may be utilized.

FIG. 2 is a block diagram of a computing system (either a server orclient, or both, as appropriate), with optional input devices (e.g.,keyboard, mouse, touch screen, etc.) and output devices, hardware,network connections, one or more processors, and memory/storage for dataand modules, etc. which may be utilized as controller and display inconjunction with embodiments of the present invention.

FIGS. 3A-L are drawings of hardware components of the surgicalmonitoring system according to embodiments of the invention.

FIGS. 4A-C is a flow chart diagram illustrating one embodiment of theregistering method of the present invention.

FIG. 5 is a drawing of a dental fiducial key with a tracking pole and adental drill according to one embodiment of the present invention.

FIG. 6 is a drawing of an endoscopic surgical site showing the fiducialkey, endoscope, and biopsy needle according to another embodiment of theinvention.

FIG. 7 is a drawing of a tracking marker bearing a pattern tag accordingto an embodiment of the present invention.

FIG. 8 is a drawing of tracking marker bearing two pattern tagsaccording to another embodiment of the present invention.

FIG. 9 is a drawing of tracking marker bearing two pattern tagsaccording to a further embodiment of the present invention.

FIG. 10 is a drawing of tracking marker bearing two pattern tagsaccording to yet a further embodiment of the present invention.

FIG. 11 is a drawing of a flow chart for a method of establishing acoordinate system at a fiducial key according to an embodiment of thepresent invention.

FIG. 12 is a drawing of a three-dimensional position and orientationtracking system according to an embodiment of the present invention.

FIG. 13 is a drawing of a three-dimensional position and orientationtracking system according to another embodiment of the presentinvention.

FIG. 14 is a drawing of a flow chart describing a method for tracking anitem bearing a pattern tag.

FIG. 15 is a drawing of a pattern on tracking marker of the presentinvention confined by a contrasting background with a parallelogramshape.

FIG. 16 is a drawing of a pattern on tracking marker of the presentinvention confined by a contrasting background with a triangular shape.

FIG. 17 is a drawing of an unconfined pattern on a tracking marker ofthe present invention.

FIG. 18 is a drawing of an unconfined pattern with a random distributionon a tracking marker of the present invention.

FIG. 19 is a drawing of a tracking marker bearing a pattern tagaccording to another embodiment of the present invention.

Corresponding reference characters indicate corresponding partsthroughout the several views. Although the drawings representembodiments of the present invention, the drawings are not necessarilyto scale and certain features may be exaggerated in order to betterillustrate and explain the present invention. The flow charts and screenshots are also representative in nature, and actual embodiments of theinvention may include further features or steps not shown in thedrawings. The exemplification set out herein illustrates an embodimentof the invention, in one form, and such exemplifications are not to beconstrued as limiting the scope of the invention in any manner.

DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

The embodiments disclosed below are not intended to be exhaustive orlimit the invention to the precise form disclosed in the followingdetailed description. Rather, the embodiments are chosen and describedso that others skilled in the art may utilize their teachings.

The detailed descriptions that follow are presented in part in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory representing alphanumeric characters or otherinformation. The hardware components are shown with particular shapesand relative orientations and sizes using particular scanningtechniques, although in the general case one of ordinary skillrecognizes that a variety of particular shapes and orientations andscanning methodologies may be used within the teaching of the presentinvention. A computer generally includes a processor for executinginstructions and memory for storing instructions and data, includinginterfaces to obtain and process imaging data. When a general-purposecomputer has a series of machine encoded instructions stored in itsmemory, the computer operating on such encoded instructions may become aspecific type of machine, namely a computer particularly configured toperform the operations embodied by the series of instructions. Some ofthe instructions may be adapted to produce signals that controloperation of other machines and thus may operate through those controlsignals to transform materials far removed from the computer itself.These descriptions and representations are the means used by thoseskilled in the art of data processing arts to most effectively conveythe substance of their work to others skilled in the art.

An algorithm is here, and generally, conceived to be a self-consistentsequence of steps leading to a desired result. These steps are thoserequiring physical manipulations of physical quantities, observing andmeasuring scanned data representative of matter around the surgicalsite. Usually, though not necessarily, these quantities take the form ofelectrical or magnetic pulses or signals capable of being stored,transferred, transformed, combined, compared, and otherwise manipulated.It proves convenient at times, principally for reasons of common usage,to refer to these signals as bits, values, symbols, characters, displaydata, terms, numbers, or the like as a reference to the physical itemsor manifestations in which such signals are embodied or expressed tocapture the underlying data of an image. It should be borne in mind,however, that all of these and similar terms are to be associated withthe appropriate physical quantities and are merely used here asconvenient labels applied to these quantities.

Some algorithms may use data structures for both inputting informationand producing the desired result. Data structures greatly facilitatedata management by data processing systems, and are not accessibleexcept through sophisticated software systems. Data structures are notthe information content of a memory, rather they represent specificelectronic structural elements that impart or manifest a physicalorganization on the information stored in memory. More than mereabstraction, the data structures are specific electrical or magneticstructural elements in memory, which simultaneously represent complexdata accurately, often data modeling physical characteristics of relateditems, and provide increased efficiency in computer operation.

Further, the manipulations performed are often referred to in terms,such as comparing or adding, commonly associated with mental operationsperformed by a human operator. No such capability of a human operator isnecessary, or desirable in most cases, in any of the operationsdescribed herein that form part of the present invention; the operationsare machine operations. Useful machines for performing the operations ofthe present invention include general-purpose digital computers or othersimilar devices. In all cases the distinction between the methodoperations in operating a computer and the method of computation itselfshould be recognized. The present invention relates to a method andapparatus for operating a computer in processing electrical or other(e.g., mechanical, chemical) physical signals to generate other desiredphysical manifestations or signals. The computer operates on softwaremodules, which are collections of signals stored on a media thatrepresents a series of machine instructions that enable the computerprocessor to perform the machine instructions that implement thealgorithmic steps. Such machine instructions may be the actual computercode the processor interprets to implement the instructions, oralternatively may be a higher level coding of the instructions that isinterpreted to obtain the actual computer code. The software module mayalso include a hardware component, wherein some aspects of the algorithmare performed by the circuitry itself rather as a result of aninstruction.

The present invention also relates to an apparatus for performing theseoperations. This apparatus may be specifically constructed for therequired purposes or it may comprise a general-purpose computer asselectively activated or reconfigured by a computer program stored inthe computer. The algorithms presented herein are not inherently relatedto any particular computer or other apparatus unless explicitlyindicated as requiring particular hardware. In some cases, the computerprograms may communicate or relate to other programs or equipmentsthrough signals configured to particular protocols, which may or may notrequire specific hardware or programming to interact. In particular,various general-purpose machines may be used with programs written inaccordance with the teachings herein, or it may prove more convenient toconstruct more specialized apparatus to perform the required methodsteps. The required structure for a variety of these machines willappear from the description below.

The present invention may deal with “object-oriented” software, andparticularly with an “object-oriented” operating system. The“object-oriented” software is organized into “objects”, each comprisinga block of computer instructions describing various procedures(“methods”) to be performed in response to “messages” sent to the objector “events” which occur with the object. Such operations include, forexample, the manipulation of variables, the activation of an object byan external event, and the transmission of one or more messages to otherobjects. Often, but not necessarily, a physical object has acorresponding software object that may collect and transmit observeddata from the physical device to the software system. Such observed datamay be accessed from the physical object and/or the software objectmerely as an item of convenience; therefore where “actual data” is usedin the following description, such “actual data” may be from theinstrument itself or from the corresponding software object or module.

Messages are sent and received between objects having certain functionsand knowledge to carry out processes. Messages are generated in responseto user instructions, for example, by a user activating an icon with a“mouse” pointer generating an event. Also, messages may be generated byan object in response to the receipt of a message. When one of theobjects receives a message, the object carries out an operation (amessage procedure) corresponding to the message and, if necessary,returns a result of the operation. Each object has a region whereinternal states (instance variables) of the object itself are stored andwhere the other objects are not allowed to access. One feature of theobject-oriented system is inheritance. For example, an object fordrawing a “circle” on a display may inherit functions and knowledge fromanother object for drawing a “shape” on a display.

A programmer “programs” in an object-oriented programming language bywriting individual blocks of code each of which creates an object bydefining its methods. A collection of such objects adapted tocommunicate with one another by means of messages comprises anobject-oriented program. Object-oriented computer programmingfacilitates the modeling of interactive systems in that each componentof the system may be modeled with an object, the behavior of eachcomponent being simulated by the methods of its corresponding object,and the interactions between components being simulated by messagestransmitted between objects.

An operator may stimulate a collection of interrelated objectscomprising an object-oriented program by sending a message to one of theobjects. The receipt of the message may cause the object to respond bycarrying out predetermined functions, which may include sendingadditional messages to one or more other objects. The other objects mayin turn carry out additional functions in response to the messages theyreceive, including sending still more messages. In this manner,sequences of message and response may continue indefinitely or may cometo an end when all messages have been responded to and no new messagesare being sent. When modeling systems utilizing an object-orientedlanguage, a programmer need only think in terms of how each component ofa modeled system responds to a stimulus and not in terms of the sequenceof operations to be performed in response to some stimulus. Suchsequence of operations naturally flows out of the interactions betweenthe objects in response to the stimulus and need not be preordained bythe programmer.

Although object-oriented programming makes simulation of systems ofinterrelated components more intuitive, the operation of anobject-oriented program is often difficult to understand because thesequence of operations carried out by an object-oriented program isusually not immediately apparent from a software listing as in the casefor sequentially organized programs. Nor is it easy to determine how anobject-oriented program works through observation of the readilyapparent manifestations of its operation. Most of the operations carriedout by a computer in response to a program are “invisible” to anobserver since only a relatively few steps in a program typicallyproduce an observable computer output.

In the following description, several terms that are used frequentlyhave specialized meanings in the present context. The term “object”relates to a set of computer instructions and associated data, which maybe activated directly or indirectly by the user. The terms “windowingenvironment”, “running in windows”, and “object oriented operatingsystem” are used to denote a computer user interface in whichinformation is manipulated and displayed on a video display such aswithin bounded regions on a raster scanned video display. The terms“network”, “local area network”, “LAN”, “wide area network”, or “WAN”mean two or more computers that are connected in such a manner thatmessages may be transmitted between the computers. In such computernetworks, typically one or more computers operate as a “server”, acomputer with large storage devices such as hard disk drives andcommunication hardware to operate peripheral devices such as printers ormodems. Other computers, termed “workstations”, provide a user interfaceso that users of computer networks may access the network resources,such as shared data files, common peripheral devices, andinter-workstation communication. Users activate computer programs ornetwork resources to create “processes” which include both the generaloperation of the computer program along with specific operatingcharacteristics determined by input variables and its environment.Similar to a process is an agent (sometimes called an intelligentagent), which is a process that gathers information or performs someother service without user intervention and on some regular schedule.Typically, an agent, using parameters typically provided by the user,searches locations either on the host machine or at some other point ona network, gathers the information relevant to the purpose of the agent,and presents it to the user on a periodic basis.

The term “desktop” means a specific user interface which presents a menuor display of objects with associated settings for the user associatedwith the desktop. When the desktop accesses a network resource, whichtypically requires an application program to execute on the remoteserver, the desktop calls an Application Program Interface, or “API”, toallow the user to provide commands to the network resource and observeany output. The term “Browser” refers to a program which is notnecessarily apparent to the user, but which is responsible fortransmitting messages between the desktop and the network server and fordisplaying and interacting with the network user. Browsers are designedto utilize a communications protocol for transmission of text andgraphic information over a worldwide network of computers, namely the“World Wide Web” or simply the “Web”. Examples of Browsers compatiblewith the present invention include the Internet Explorer program sold byMicrosoft Corporation (Internet Explorer is a trademark of MicrosoftCorporation), the Opera Browser program created by Opera Software ASA,or the Firefox browser program distributed by the Mozilla Foundation(Firefox is a registered trademark of the Mozilla Foundation). Althoughthe following description details such operations in terms of a graphicuser interface of a Browser, the present invention may be practiced withtext based interfaces, or even with voice or visually activatedinterfaces, that have many of the functions of a graphic based Browser.

Browsers display information, which is formatted in a StandardGeneralized Markup Language (“SGML”) or a HyperText Markup Language(“HTML”), both being scripting languages, which embed non-visual codesin a text document through the use of special ASCII text codes. Files inthese formats may be easily transmitted across computer networks,including global information networks like the Internet, and allow theBrowsers to display text, images, and play audio and video recordings.The Web utilizes these data file formats to conjunction with itscommunication protocol to transmit such information between servers andworkstations. Browsers may also be programmed to display informationprovided in an eXtensible Markup Language (“XML”) file, with XML filesbeing capable of use with several Document Type Definitions (“DTD”) andthus more general in nature than SGML or HTML. The XML file may beanalogized to an object, as the data and the stylesheet formatting areseparately contained (formatting may be thought of as methods ofdisplaying information, thus an XML file has data and an associatedmethod).

The terms “personal digital assistant” or “PDA”, as defined above, meansany handheld, mobile device that combines computing, telephone, fax,e-mail and networking features. The terms “wireless wide area network”or “WWAN” mean a wireless network that serves as the medium for thetransmission of data between a handheld device and a computer. The term“synchronization” means the exchanging of information between a firstdevice, e.g. a handheld device, and a second device, e.g. a desktopcomputer, either via wires or wirelessly. Synchronization ensures thatthe data on both devices are identical (at least at the time ofsynchronization).

In wireless wide area networks, communication primarily occurs throughthe transmission of radio signals over analog, digital cellular, orpersonal communications service (“PCS”) networks. Signals may also betransmitted through microwaves and other electromagnetic waves. At thepresent time, most wireless data communication takes place acrosscellular systems using second generation technology such ascode-division multiple access (“CDMA”), time division multiple access(“TDMA”), the Global System for Mobile Communications (“GSM”), ThirdGeneration (wideband or “3G”), Fourth Generation (broadband or “4G”),personal digital cellular (“PDC”), or through packet-data technologyover analog systems such as cellular digital packet data (CDPD”) used onthe Advance Mobile Phone Service (“AMPS”).

The terms “wireless application protocol” or “WAP” mean a universalspecification to facilitate the delivery and presentation of web-baseddata on handheld and mobile devices with small user interfaces. “MobileSoftware” refers to the software operating system, which allows forapplication programs to be implemented on a mobile device such as amobile telephone or PDA. Examples of Mobile Software are Java and JavaME (Java and JavaME are trademarks of Sun Microsystems, Inc. of SantaClara, Calif.), BREW (BREW is a registered trademark of QualcommIncorporated of San Diego, Calif.), Windows Mobile (Windows is aregistered trademark of Microsoft Corporation of Redmond, Wash.), PalmOS (Palm is a registered trademark of Palm, Inc. of Sunnyvale, Calif.),Symbian OS (Symbian is a registered trademark of Symbian SoftwareLimited Corporation of London, United Kingdom), ANDROID OS (ANDROID is aregistered trademark of Google, Inc. of Mountain View, Calif.), andiPhone OS (iPhone is a registered trademark of Apple, Inc. of Cupertino,Calif.), and Windows Phone 7. “Mobile Apps” refers to software programswritten for execution with Mobile Software.

The terms “scan,” “fiducial reference”, “fiducial location”, “marker,”“tracker” and “image information” have particular meanings in thepresent disclosure. For purposes of the present disclosure, “scan” orderivatives thereof refer to x-ray, magnetic resonance imaging (MRI),computerized tomography (CT), sonography, cone beam computerizedtomography (CBCT), or any system that produces a quantitative spatialrepresentation of a patient. The term “fiducial reference” or simply“fiducial” refers to an object or reference on the image of a scan thatis uniquely identifiable as a fixed recognizable point. In the presentspecification the term “fiducial location” refers to a useful locationto which a fiducial reference is attached. A “fiducial location” willtypically be proximate a surgical site. The term “marker” or “trackingmarker” refers to an object or reference that may be perceived by asensor proximate to the location of the surgical or dental procedure,where the sensor may be an optical sensor, a radio frequency identifier(RFID), a sonic motion detector, an ultra-violet or infrared sensor. Theterm “tracker” refers to a device or system of devices able to determinethe location of the markers and their orientation and movementcontinually in ‘real time’ during a procedure. As an example of apossible implementation, if the markers are composed of printed targetsthen the tracker may include a stereo camera pair. The term “imageinformation” is used in the present specification to describeinformation obtained by the tracker, whether optical or otherwise, andusable for determining the location of the markers and their orientationand movement continually in ‘real time’ during a procedure.

FIG. 1 is a high-level block diagram of a computing environment 100according to one embodiment. FIG. 1 illustrates server 110 and threeclients 112 connected by network 114. Only three clients 112 are shownin FIG. 1 in order to simplify and clarify the description. Embodimentsof the computing environment 100 may have thousands or millions ofclients 112 connected to network 114, for example the Internet. Users(not shown) may operate software 116 on one of clients 112 to both sendand receive messages network 114 via server 110 and its associatedcommunications equipment and software (not shown).

FIG. 2 depicts a block diagram of computer system 210 suitable forimplementing server 110 or client 112. Computer system 210 includes bus212 which interconnects major subsystems of computer system 210, such ascentral processor 214, system memory 217 (typically RAM, but which mayalso include ROM, flash RAM, or the like), input/output controller 218,external audio device, such as speaker system 220 via audio outputinterface 222, external device, such as display screen 224 via displayadapter 226, serial ports 228 and 230, keyboard 232 (interfaced withkeyboard controller 233), storage interface 234, disk drive 237operative to receive floppy disk 238, host bus adapter (HBA) interfacecard 235A operative to connect with Fibre Channel network 290, host busadapter (HBA) interface card 235B operative to connect to SCSI bus 239,and optical disk drive 240 operative to receive optical disk 242. Alsoincluded are mouse 246 (or other point-and-click device, coupled to bus212 via serial port 228), modem 247 (coupled to bus 212 via serial port230), and network interface 248 (coupled directly to bus 212).

Bus 212 allows data communication between central processor 214 andsystem memory 217, which may include read-only memory (ROM) or flashmemory (neither shown), and random access memory (RAM) (not shown), aspreviously noted. RAM is generally the main memory into which operatingsystem and application programs are loaded. ROM or flash memory maycontain, among other software code, Basic Input-Output system (BIOS),which controls basic hardware operation such as interaction withperipheral components. Applications resident with computer system 210are generally stored on and accessed via computer readable media, suchas hard disk drives (e.g., fixed disk 244), optical drives (e.g.,optical drive 240), floppy disk unit 237, or other storage medium.Additionally, applications may be in the form of electronic signalsmodulated in accordance with the application and data communicationtechnology when accessed via network modem 247 or interface 248 or othertelecommunications equipment (not shown).

Storage interface 234, as with other storage interfaces of computersystem 210, may connect to standard computer readable media for storageand/or retrieval of information, such as fixed disk drive 244. Fixeddisk drive 244 may be part of computer system 210 or may be separate andaccessed through other interface systems. Modem 247 may provide directconnection to remote servers via telephone link or the Internet via anInternet service provider (ISP) (not shown). Network interface 248 mayprovide direct connection to remote servers via direct network link tothe Internet via a POP (point of presence). Network interface 248 mayprovide such connection using wireless techniques, including digitalcellular telephone connection, Cellular Digital Packet Data (CDPD)connection, digital satellite data connection or the like.

Many other devices or subsystems (not shown) may be connected in asimilar manner (e.g., document scanners, digital cameras and so on),including the hardware components of FIGS. 3A-I, which alternatively maybe in communication with associated computational resources throughlocal, wide-area, or wireless networks or communications systems. Thus,while the disclosure may generally discuss an embodiment where thehardware components are directly connected to computing resources, oneof ordinary skill in this area recognizes that such hardware may beremotely connected with computing resources. Conversely, all of thedevices shown in FIG. 2 need not be present to practice the presentdisclosure. Devices and subsystems may be interconnected in differentways from that shown in FIG. 2. Operation of a computer system such asthat shown in FIG. 2 is readily known in the art and is not discussed indetail in this application. Software source and/or object codes toimplement the present disclosure may be stored in computer-readablestorage media such as one or more of system memory 217, fixed disk 244,optical disk 242, or floppy disk 238. The operating system provided oncomputer system 210 may be a variety or version of either MS-DOS®(MS-DOS is a registered trademark of Microsoft Corporation of Redmond,Wash.), WINDOWS® (WINDOWS is a registered trademark of MicrosoftCorporation of Redmond, Wash.), OS/2® (OS/2 is a registered trademark ofInternational Business Machines Corporation of Armonk, N.Y.), UNIX®(UNIX is a registered trademark of X/Open Company Limited of Reading,United Kingdom), Linux® (Linux is a registered trademark of LinusTorvalds of Portland, Oreg.), or other known or developed operatingsystem.

Moreover, regarding the signals described herein, those skilled in theart recognize that a signal may be directly transmitted from a firstblock to a second block, or a signal may be modified (e.g., amplified,attenuated, delayed, latched, buffered, inverted, filtered, or otherwisemodified) between blocks. Although the signals of the above-describedembodiments are characterized as transmitted from one block to the next,other embodiments of the present disclosure may include modified signalsin place of such directly transmitted signals as long as theinformational and/or functional aspect of the signal is transmittedbetween blocks. To some extent, a signal input at a second block may beconceptualized as a second signal derived from a first signal outputfrom a first block due to physical limitations of the circuitry involved(e.g., there will inevitably be some attenuation and delay). Therefore,as used herein, a second signal derived from a first signal includes thefirst signal or any modifications to the first signal, whether due tocircuit limitations or due to passage through other circuit elementswhich do not change the informational and/or final functional aspect ofthe first signal.

The present invention relates to a surgical hardware and softwaremonitoring system and method which allows for surgical planning whilethe patient is available for surgery, for example while the patient isbeing prepared for surgery so that the system may model the surgicalsite. The system uses a particularly configured piece of hardware,represented as fiducial key 10 in FIG. 3A, to orient tracking marker 12of the monitoring system with regard to the critical area of thesurgery. Fiducial key 10 is attached to a location near the intendedsurgical area, in the exemplary embodiment of the dental surgical areaof FIG. 3A, fiducial key 10 is attached to a dental splint 14. Trackingmarker 12 may be connected to fiducial key 10 by tracking pole 11. Inembodiments in which the fiducial reference is directly visible to asuitable tracker (see for example FIG. 5 and FIG. 6) that acquires imageinformation about the surgical site, a tracking marker may be attacheddirectly to the fiducial reference. For example a dental surgery, thedental tracking marker 14 may be used to securely locate the fiducial 10near the surgical area. The fiducial key 10 may be used as a point ofreference, or a fiducial, for the further image processing of dataacquired from tracking marker 12 by the tracker.

In other embodiments additional tracking markers 12 may be attached toitems independent of the fiducial key 10 and any of its associatedtracking poles 11 or tracking markers 12. This allows the independentitems to be tracked by the tracker.

In a further embodiment at least one of the items or instruments nearthe surgical site may optionally have a tracker attached to function astracker for the monitoring system of the invention and to thereby sensethe orientation and the position of the tracking marker 12 and of anyother additional tracking markers relative to the scan data of thesurgical area. By way of example, the tracker attached to an instrumentmay be a miniature digital camera and it may be attached, for example,to a dentist's drill. Any other markers to be tracked by the trackerattached to the item or instrument must be within the field of view ofthe tracker.

Using the dental surgery example, the patient is scanned to obtain aninitial scan of the surgical site. The particular configuration offiducial key 10 allows computer software stored in memory and executedin a suitable controller, for example processor 214 and memory 217 ofcomputer 210 of FIG. 2, to recognize its relative position within thesurgical site from the scan data, so that further observations may bemade with reference to both the location and orientation of fiducial key10. In some embodiments, the fiducial reference includes a marking thatis apparent as a recognizable identifying symbol when scanned. In otherembodiments, the fiducial reference includes a shape that is distinct inthe sense that the body apparent on the scan has an asymmetrical formallowing the front, rear, upper, and lower, and left/right definedsurfaces that may be unambiguously determined from the analysis of thescan, thereby to allow the determination not only of the location of thefiducial reference, but also of its orientation.

In addition, the computer software may create a coordinate system fororganizing objects in the scan, such as teeth, jaw bone, skin and gumtissue, other surgical instruments, etc. The coordinate system relatesthe images on the scan to the space around the fiducial and locates theinstruments bearing tracking markers both by orientation and position.The model generated by the monitoring system may then be used to checkboundary conditions, and in conjunction with the tracker display thearrangement in real time on a suitable display, for example display 224of FIG. 2.

In one embodiment, the computer system has a predetermined knowledge ofthe physical configuration of fiducial key 10 and examinesslices/sections of the scan to locate fiducial key 10. Locating offiducial key 10 may be on the basis of its distinct shape, or on thebasis of distinctive identifying and orienting markings upon thefiducial key or on attachments to the fiducial key 10 as tracking marker12. Fiducial key 10 may be rendered distinctly visible in the scansthrough higher imaging contrast by the employ of radio-opaque materialsor high-density materials in the construction of the fiducial key 10. Inother embodiments the material of the distinctive identifying andorienting markings may be created using suitable high density orradio-opaque inks or materials.

Once fiducial key 10 is identified, the location and orientation of thefiducial key 10 is determined from the scan segments, and a point withinfiducial key 10 is assigned as the center of the coordinate system. Thepoint so chosen may be chosen arbitrarily, or the choice may be based onsome useful criterion. A model is then derived in the form of atransformation matrix to relate the fiducial system, being fiducial key10 in one particular embodiment, to the coordinate system of thesurgical site. The resulting virtual construct may be used by surgicalprocedure planning software for virtual modeling of the contemplatedprocedure, and may alternatively be used by instrumentation software forthe configuration of the instrument, for providing imaging assistancefor surgical software, and/or for plotting trajectories for the conductof the surgical procedure.

In some embodiments, the monitoring hardware includes a trackingattachment to the fiducial reference. In the embodiment pertaining todental surgery the tracking attachment to fiducial key 10 is trackingmarker 12, which is attached to fiducial key 10 via tracking pole 11.Tracking marker 12 may have a particular identifying pattern. Thetrackable attachment, for example tracking marker 12, and evenassociated tracking pole 11 may have known configurations so thatobservational data from tracking pole 11 and/or tracking marker 12 maybe precisely mapped to the coordinate system, and thus progress of thesurgical procedure may be monitored and recorded. For example, asparticularly shown in FIG. 3J, fiducial key 10 may have hole 15 in apredetermined location specially adapted for engagement with insert 17of tracking pole 11. In such an arrangement, for example, tracking poles11 may be attached with a low force push into hole 15 of fiducial key10, and an audible haptic notification may thus be given upon successfulcompletion of the attachment.

It is further possible to reorient the tracking pole during a surgicalprocedure. Such reorientation may be in order to change the location ofthe procedure, for example where a dental surgery deals with teeth onthe opposite side of the mouth, where a surgeon switches hands, and/orwhere a second surgeon performs a portion of the procedure. For example,the movement of the tracking pole may trigger a re-registration of thetracking pole with relation to the coordinate system, so that thelocations may be accordingly adjusted. Such a re-registration may beautomatically initiated when, for example in the case of the dentalsurgery embodiment, tracking pole 11 with its attached tracking marker12 are removed from hole 15 of fiducial key 10 and another trackingmarker with its associated tracking pole is connected to an alternativehole on fiducial key 10. Additionally, boundary conditions may beimplemented in the software so that the user is notified whenobservational data approaches and/or enters the boundary areas.

In a further embodiment, the tracking markers may specifically have athree dimensional shape. Suitable three-dimensional shapes bearingidentifying patterns may include, without limitation, a segment of anellipsoid surface and a segment of a cylindrical surface. In general,suitable three-dimensional shapes are shapes that are mathematicallydescribable by simple functions. One particular three-dimensionalsurface suitable for use as tracking marker 312 in this embodiment is acylindrical surface, as shown in FIG. 3K. A cylindrical surface ismathematically described by a simple function. Pattern 313 isrotationally asymmetric, so that rotating cylindrically shaped trackingmarker 312 never causes pattern 313 to repeat itself spatially. Thisallows the position and orientation of tracking marker 312 to beuniquely determined. Pattern 313 may be present over any useful segmentof the surface of tracking marker 312, and may extend around the fullcircular perimeter of tracking marker 312, thereby allowing a suitabletracker (not shown) to always have a portion of pattern 313 in its view,irrespective of the orientation of position of tracking marker 312.Tracking marker 312 may engage with tracking pole 11 in exactly the sameway as already described in the case of tracking markers 12. In FIG. 3K,tracking marker 312 is shown as comprising of five rings of patternswhich, together, comprise pattern 313. In other embodiments trackingmarker 312 may comprise a single ring bearing a suitably rotationallyasymmetric pattern 313 and tracking marker 312 may thereby be a simplering bearing pattern 313.

Further embodiments of suitable tracking markers bearing rotationallyasymmetric patterns are described later at the hand of FIGS. 7-10. Thecontrast aspects discussed below at the hand of FIGS. 7-10 also apply topattern 313 in FIG. 3K in that the contrasting portions of pattern 313may have perimeters comprising a mathematically describable curvedsections to provide suitable pattern tags. More detail in this regard isprovided below.

In another embodiment, a suitable segment of a three-dimensional surfacefor use as a pattern bearing surface for a tracking marker is anellipsoid surface. Ellipsoids are describable by simple mathematicalfunctions, of which a spherical surface is the most simple and thesurface extends in three dimensions (i.e. not flat and extending in onlytwo dimensions). FIG. 3L shows tracking marker 322 having an ellipsoidsurface bearing a pattern 323. Tracking marker 322 may be used in thesame fashion as tracking marker 312, or the tracking markers of FIGS.7-10, 15, 16, 17, and 18.

In both FIG. 3K and FIG. 3L patterns 313 and 323 respectively are shownas black circular areas on a white background. In other embodiments, thecircular contrast areas may be white and the background color may beblack. In yet further embodiments, other mutually contrasting colors,tones, or hues may be employed in the visible or infrared spectrum forthe circular contrast areas and for the background, as described laterbelow at the hand of FIG. 7.

The tracker of the system may comprise a single imager obtaining atwo-dimensional image of the site being monitored. The system and methoddescribed in the present specification allow three-dimensional locationsand orientations of tracking markers to be obtained usingnon-stereo-pair two-dimensional imagery. In some embodiments more thanone imager may be employed as tracker, but the image informationrequired and employed is nevertheless two-dimensional. Therefore the twoimagers may merely be employed to secure different perspective views ofthe site, each imager rendering a two-dimensional image that is not partof a stereo pair. This does not exclude the employment of stereo-imagersin obtaining the image information about the site, but the system andmethod are not reliant on stereo imagery of the site.

In a further embodiment of the system utilizing the invention, asurgical instrument or implement, herein termed a “hand piece” (seeFIGS. 5 and 6), may also have a particular configuration that may belocated and tracked in the coordinate system and may have suitabletracking markers as described herein. A boundary condition may be set upto indicate a potential collision with virtual material, so that whenthe hand piece is sensed to approach the boundary condition anindication may appear on a screen, or an alarm sound. Further, targetboundary conditions may be set up to indicate the desired surgical area,so that when the trajectory of the hand piece is trending outside thetarget area an indication may appear on screen or an alarm soundindicating that the hand piece is deviating from its desired path.

An alternative embodiment of some hardware components are shown in FIGS.3G-I. Fiducial key 10′ has connection elements with suitable connectingportions to allow a tracking pole 11′ to position a tracking marker 12′relative to the surgical site. Conceptually, fiducial key 10′ serves asan anchor for pole 11′ and tracking marker 12′ in much the same way asthe earlier embodiment, although it has a distinct shape. The softwareof the monitoring system is pre-programmed with the configuration ofeach particularly identified fiducial key, tracking pole, and trackingmarker, so that the location calculations are only changed according tothe changed configuration parameters.

The materials of the hardware components may vary according toregulatory requirements and practical considerations. Generally, the keyor fiducial component is made of generally radio opaque material suchthat it does not produce noise for the scan, yet creates recognizablecontrast on the scanned image so that any identifying pattern associatedwith it may be recognized. In addition, because it is generally locatedon the patient, the material should be lightweight and suitable forconnection to an apparatus on the patient. For example, in the dentalsurgery example, the materials of the fiducial key must be suitable forconnection to a plastic splint and suitable for connection to a trackingpole. In the surgical example the materials of the fiducial key may besuitable for attachment to the skin or other particular tissue of apatient.

The tracking markers are clearly identified by employing, for examplewithout limitation, high contrast pattern engraving. The materials ofthe tracking markers are chosen to be capable of resisting damage inautoclave processes and are compatible with rigid, repeatable, and quickconnection to a connector structure. The tracking markers and associatedtracking poles have the ability to be accommodated at differentlocations for different surgery locations, and, like the fiducial keys,they should also be relatively lightweight as they will often be restingon or against the patient. The tracking poles must similarly becompatible with autoclave processes and have connectors of a form sharedamong tracking poles.

The tracker employed in tracking the fiducial keys, tracking poles andtracking markers should be capable of tracking with suitable accuracyobjects of a size of the order of 1.5 square centimeters. The trackermay be, by way of example without limitation, a stereo camera or stereocamera pair. While the tracker is generally connected by wire to acomputing device to read the sensory input, it may optionally havewireless connectivity to transmit the sensory data to a computingdevice.

In embodiments that additionally employ a trackable piece ofinstrumentation, such as a hand piece, tracking markers attached to sucha trackable piece of instrumentation may also be light-weight; capableof operating in a 3-object array with 90 degrees relationship;optionally having a high contrast pattern engraving and a rigid, quickmounting mechanism to a standard hand piece.

In another aspect there is presented an automatic registration methodfor tracking surgical activity, as illustrated in FIGS. 4A-C. FIG. 4Aand FIG. 4B together present, without limitation, a flowchart of onemethod for determining the three-dimensional location and orientation ofthe fiducial reference from scan data. FIG. 4C presents a flow chart ofa method for confirming the presence of a suitable tracking marker inimage information obtained by the tracker and determining thethree-dimensional location and orientation of the fiducial referencebased on the image information.

Once the process starts [402], as described in FIGS. 4A and 4B, thesystem obtains a scan data set [404] from, for example, a CT scanner andchecks for a default CT scan Hounsfield unit (HU) value [at 406] for thefiducial which may or may not have been provided with the scan based ona knowledge of the fiducial and the particular scanner model, and ifsuch a threshold value is not present, then a generalized predetermineddefault value is employed [408]. Next the data is processed by removingscan segments with Hounsfield data values outside expected valuesassociated with the fiducial key values [at 410], following thecollection of the remaining points [at 412]. If the data is empty [at414], the CT value threshold is adjusted [at 416], the original valuerestored [at 418], and the segmenting processing scan segments continues[at 410]. Otherwise, with the existing data a center of mass iscalculated [at 420], along with calculating the X, Y, and Z axes [at422]. If the center of mass is not at the cross point of the XYZ axes[at 424], then the user is notified [at 426] and the process stopped [at428]. If the center of mass is at the XYZ cross point then the datapoints are compared with the designed fiducial data [430]. If thecumulative error is larger than the maximum allowed error [432] then theuser is notified [at 434] and the process ends [at 436]. If not, thenthe coordinate system is defined at the XYZ cross point [at 438], andthe scan profile is updated for the HU units [at 440].

Turning now to FIG. 4C, image information is obtained from the tracker,being a suitable camera or other sensor [442]. The image information istwo-dimensional and is not required to be a stereo image pair. The imageinformation may be sourced from a single imaging device in the tracker,or may be sourced from multiple imaging devices in the tracker. It bearspointing out that the presence of multiple imaging devices in a trackerdoes not automatically imply stereo imaging. The image information isanalyzed to determine whether a tracking marker is present in the imageinformation [444]. If not, then the user is queried [446] as to whetherthe process should continue or not. If not, then the process is ended[448]. If the process is to continue, then the user can be notified thatno tracking marker has been found in the image information [450], andthe process returns to obtaining image information [442]. If a trackingmarker has been found based on the image information, or one has beenattached by the user upon the above notification [450], the offset andrelative orientation of the tracking marker to the fiducial reference isobtained from a suitable database [452]. The term “database” is used inthis specification to describe any source, amount or arrangement of suchinformation, whether organized into a formal multi-element ormulti-dimensional database or not. A single data set comprising offsetvalue and relative orientation may suffice in a simple implementation ofthis embodiment of the invention and may be provided, for example, bythe user or may be within a memory unit of the controller or in aseparate database or memory.

The offset and relative orientation of the tracking marker is used todefine the origin of a coordinate system at the fiducial reference andto determine the three-dimensional orientation of the fiducial referencebased on the image information [454] and the registration process ends[458]. In order to monitor the location and orientation of the fiducialreference in real time, the process may be looped back from step [454]to obtain new image information from the camera [442]. A suitable querypoint may be included to allow the user to terminate the process.Detailed methods for determining orientations and locations ofpredetermined shapes or marked tracking markers from image data areknown to practitioners of the art and will not be dwelt upon here. Thecoordinate system so derived is then used for tracking the motion of anyitems bearing tracking markers in the proximity of the surgical site.Other registration systems are also contemplated, for example usingcurrent other sensory data rather than the predetermined offset, orhaving a fiducial with a transmission capacity.

One example of an embodiment of the invention is shown in FIG. 5. Inaddition to fiducial key 502 mounted at a predetermined tooth and havinga rigidly mounted tracking marker 504, an additional instrument orimplement 506, for example a hand piece which may be a dental drill, maybe observed by a camera 508 serving as tracker of the monitoring system.

Another example of an embodiment of the invention is shown in FIG. 6.Surgery site 600, for example a human stomach or chest, may havefiducial key 602 fixed to a predetermined position to support trackingmarker 604. Endoscope 606 may have further tracking markers, and biopsyneedle 608 may also be present bearing a tracking marker at surgery site600. Sensor 610, may be for example a camera, infrared sensing device,or RADAR.

A further aspect of the invention is described at the hand of FIG. 7,which shows in more detail tracking marker 12 of FIGS. 3A and 3B. Asstated heretofore, tracking marker 12 may have a particular identifyingpattern. In this further aspect of the invention the matter of theparticular pattern, shown generally at 72, on tracking marker 12 isaddressed in more detail. In a first embodiment shown in FIG. 7 thepattern 72 comprises a plurality of contrasting portions 74. Pattern 72is further characterized by being rotationally asymmetrical. As aresult, an image of the pattern 72 inherently identifies the rotationalorientation about an axis perpendicular to the plane of pattern 72 oftracking marker 12. Pattern 72 is further characterized by having atleast one contrasting portion 74 that has a perimeter comprising amathematically describable curved section. In FIG. 7 the simplest caseof a circular perimeter 76 is shown, which comprises the entireperimeter. In other embodiments the curved section may constitute lessthan the entire perimeter and the curve may be, for example, withoutlimitation, a conic section. In yet further embodiments the curve can bea mathematically describable curve other than a conic section. In yetfurther embodiments, the perimeter may be a mathematically describedpolygon.

The basis or grounds of the contrast is limited only in that thecontrast has to be discernable by the tracker employed in the surgicalsite monitoring system of the present invention. For example withoutlimitation, the contrast with surrounding areas on the tracking marker12 may be by virtue of the contrasting portion 74 being a cutout, byvirtue of the contrasting portion 74 being a darker or lighter greytone, by virtue of the contrasting portion 74 being a different hue orsaturation, by virtue of the contrasting portion 74 being a differentcolor in any color space, by virtue of the contrasting portion 74 beinga different brightness in an infrared image, or any other basis of imagecontrast.

The pattern 72 may be implemented on a separate pattern tag 77 that isattached or pasted, temporarily or permanently, to the tracking marker12. Conversely, the pattern tag 77 may be in itself a tracking marker,such as, for example tracking marker 12, so that the tracking markeritself bears pattern 72. The pattern tag 77 may be planar. The patterntag 77 may be flexible to allow it to return to planarity after beingflexibly deformed. The materials of the pattern tag 77 may be, forexample without limitation, a polymer or a paper or a mix of both paperand polymer. In other embodiments the tag 77 may be non-flexiblydeformable while remaining dimensionally stable. An individual trackingmarker may comprise a plurality of pattern tags, each with a pattern ofits own, as will be described below.

In some embodiments, the perimeter may be a mathematically describedpolygon. In other embodiments, the presence of the mathematicallydescribable curved section provides three distinct benefits. Firstly, itovercomes the inherent problem of straight-edged shapes such as squares,rectangles, and parallelograms which exacerbate problems stemming fromthe finite number and size of pixels available in typical trackers, suchas the tracker used in the several embodiments of the present invention.Due to the fact that the pixels have a finite size, the determination ofthe exact location of a straight line in an image is difficult to do toan accuracy of less than one pixel. A contrasting portion with astraight-line section to its perimeter would inherently suffer from thislimitation. By employing a mathematically describable curved section asperimeter 76 of contrasting portion 74 the location of perimeter 76 caninherently be determined more accurately. We do not dwell here upon themethods of determining contrast boundaries in digital images, as theconcepts and methods are well described in the art and well known topractioners of the art.

Secondly, in addition to the aforementioned more accurate determinationof the location of the perimeter, the mathematically describable natureof the curve of the perimeter 76 allows a single very accuratecontrasting portion reference point 78 to be determined once an image ofthe pattern 72 is available, showing its contrasting portion 74 andperimeter 76. By way of the circular example of FIG. 7, a useful choicefor a contrasting portion reference point 78 may be the center of thecircle described by perimeter 76, which in this case is the center ofthe contrasting portion 74. However, in a more general case, a pointother than the center of the circle may be employed as reference to suitthe application.

Thirdly, with the mathematical description of a section of the perimeter76 of contrasting portion 74 known, the rotation of pattern 72 aboutfurther axes may be determined To this end, the appearance of thepattern 72 may be expressed in mathematical terms and stored in adatabase of any kind, including without limitation a digital database.The tracker of the monitoring system may obtain image information aboutthe pattern 72 on a tracking marker 12. By analyzing the imageinformation mathematically using a suitable controller, for exampleprocessor 214 and memory 217 of computer 210 of FIG. 2, and comparingwith the stored information about the mathematical description of thepattern, the three-dimensional orientation of tracking marker 12 may bedetermined. If tracking marker 12 has a large enough three-dimensionalextent, then suitable patterns of contrasting portions may also beapplied to further surfaces of tracking marker 12 to assist indetermining the three-dimensional orientation of tracking marker 12. Thesame approach may be applied in embodiments in which the perimeter is amathematically described polygon.

The pattern 72 may be selected to be a unique pattern. This allows thepattern tag 77 or the tracking marker 12 to be uniquely identifiedwithin the field of view of the tracker. Thus a variety of items,objects, instruments or implements may be tagged with tracking markersbearing pattern tags, or with just pattern tags, thereby to uniquelyidentify and track such items, objects, instruments or implements anddetermine their orientations.

Having described this general aspect of the invention at the hand ofcontrasting portions with simple circular shapes, we turn to otherembodiments employing contrasting portions employing other curvedshapes. In other embodiments the curve may be, for example any otherform of conic section, such as an ellipse or a parabola and may extendall the way around the contrasting portion. In the case of an ellipse,the contrasting portion reference point may be chosen, for example, tolie along the major semi-axis or minor semi-axis of the ellipse. Inparticular, a useful choice for contrasting portion reference point maybe one of the focii of the ellipse. Another useful choice forcontrasting portion reference point may be one of the vertices of theellipse. In this respect it is to be noted that all that is required isa section of an ellipse, long enough for accurate mathematicdescription, thereby to allow the determination of the various axes andthe focii. The contrasting portion therefore does not have to be acomplete ellipse. Herein lies the benefit of the curve beingmathematically describable. If a parabola is chosen, a useful choice forcontrasting portion reference point may be the focus of the parabola,the vertex of the parabola or the point where the axis of symmetry ofthe parabola crosses the directrix of that parabola.

In yet further embodiments of the invention a mathematically describablecurve other than a conic section may be used to describe at least asection of the perimeter of the contrasting portion. Such curves maywell be more complex than conic sections and may require carefulconsideration as regards a suitable contrasting portion reference point.In yet further embodiments of the invention, the contrasting portion canbe a mix of the aforementioned conic sections and other shapes. Oneexample is a semicircle, which, despite having only part of itsperimeter described by a circle, nevertheless allows all of the benefitsof the mathematically described circle.

In yet further embodiments of the invention the pattern may comprise aplurality of contrasting portions of which more than one contrastingportion has a perimeter having a mathematically describable curvedsection or that is a mathematically describable polygon. A patternreference point may in such a case be a point expressed relative to theresulting plurality of contrasting portion reference points derived fromthe more than one contrasting portion. For example without limitation,each of the three contrasting portions of pattern tag 77 in FIG. 7 is acircle and each has its center as contrasting portion reference point.In such a case, the pattern reference point may be, for example, givenby a point exactly at the middle of the line joining the centers of thetwo unnumbered contrasting portions. Any other useful point may beselected for this purpose, including the contrasting portion referencepoint 78 or any of the corners of the pattern tag 77.

In a further implementation shown in FIG. 8, tracking marker 12 maycomprise more than one pattern tag, for example pattern tag 87 andpattern tag 87′, with each pattern tag 87 and 87′ individually having apattern shown generally at 82 and 82′ respectively and each havingrotational symmetry, while the combination of patterns 82 and 82′ isrotationally asymmetrical. In this particular implementation the twotags are identical, but, in a general case, are located on trackingmarker rotated with respect to each other. This has the benefit ofrequiring only one kind of patterned tag. It reduces costs and alsolowers the management burden during practical use, as only one kind oftag needs to be kept at hand for in, for example, surgery. In anotherembodiment, the two pattern tags 87 and 87′ may be arranged next to eachother on tracking marker 12 in identical orientations. This stillprovides a resulting pattern that is rotationally asymmetric. In FIG. 8the two pattern tags 87 and 87′ are shown as being attached in coplanarfashion. In other embodiments they are not limited to being coplanar.

In a further implementation shown in FIG. 9 two pattern tags 97 and 97′are employed and both have some form of rotational symmetry. Pattern tag97 has a pattern 92 with rotational symmetry of 120 degrees whilepattern tag 97′ has a pattern 92′ that differs from pattern 92 and has arotational symmetry of 180 degrees. The two pattern tags 97 and 97′together, however, provide rotational asymmetry.

In FIG. 10 is presented yet a further implementation based on thepattern 102 of pattern tag 107 having rotational symmetry and thepattern 102′ of pattern tag 107′ being rotationally asymmetrical. Thejoint patterns 102 and 102′ constitute a rotationally asymmetricalpattern.

In FIGS. 7-10 very simple patterns have been used as examples. Thepatterns may be chosen to be more complex and thereby more unique. Thisallows the pattern tags to be uniquely identified within the field ofview of the tracker. Thus, a variety of items, objects, instruments orimplements may be tagged with pattern tags, thereby to uniquely identifyand track such items, objects, instruments or implements and determinetheir orientations. The sets of two pattern tags of FIGS. 8-10 can ineach embodiment of the invention constitute a single tracking marker.Yet further suitable patterns are discussed below at the hand of FIGS.15 to 18.

The patterns 82, 82′, 92, 92′, 102, 102′ of FIGS. 8-10 may beimplemented on separate pattern tags that are attached or pasted,temporarily or permanently, to tracking marker 12. Conversely, the pairsof pattern tags (87, 87′), (97, 97′), and (107, 107′) may in themselvesbe tracking markers, such as, for example tracking marker 12, so thatthe tracking markers themselves bear patterns (82, 82′), (92, 92′), and(102, 102′), respectively. The pattern tags may be planar. The patterntags may be flexible to allow them to return to planarity after beingflexibly deformed. The materials of the pattern tags may be, for examplewithout limitation, a polymer or a paper, or a mix of both paper andpolymer. In other embodiments, the pattern tags may be non-flexiblydeformable while remaining dimensionally stable.

The automatic registration method for tracking surgical activity alreadydescribed at the hand of FIGS. 4A-C may employ the tracking marker ofFIGS. 7-10 bearing the pattern tags and or patterns described at thehand of FIGS. 7-10. In the method of FIG. 4A the offset and relativeorientation of the tracking marker to the fiducial reference is obtainedfrom a suitable database in method step [452]. If the tracking marker,pattern tags and patterns of FIGS. 7-10 are employed, then the databasein question is pre-supplied with information concerning the trackingmarker 12, the pattern tags 77, 87, 87′, 97, 97′, 107, 107′, thepatterns 72, 82, 82′, 92, 92′, 102, 102′ and the contrasting portions,for example contrasting portion 74, of the pattern tags. The informationcomprises, in particular, the mathematical descriptions of curvedsections or polygons of the perimeters of the contrasting portions ofthe pattern tags, for example perimeter 76. It may also comprise thelocations of contrasting portion reference points, for examplecontrasting portion reference point 78, and pattern reference points forpattern tags that are be employed. The term “geometric information” isemployed in the present specification to describe this collection ofinformation regarding the shapes, sizes, perimeters, curved perimetersections and the like of the contrasting portions of the pattern tags,along with the information on the patterns on the various pattern tagsattached to the tracking markers and the associated locations ofcontrasting portion reference points and pattern reference points. Thegeometric information specifically comprises a mathematical descriptionof at least a section of the perimeter of at least one contrastingportion on any given pattern tag. The geometric information may alsoinclude the known spatial and orientational relationship between thepattern tags and the tracking markers.

The automatic registration method for tracking surgical activity as perthe present embodiment employing the pattern tags (for example patterntag 77) as described herein comprises the steps [402] to [456] of FIGS.4A-C. In step [444] of FIG. 4C tracking marker 12 has already beenidentified on the basis of its unique pattern as per FIGS. 7-10. Step454 of FIG. 4C will now be described in more detail at the hand of FIG.11. The using [454] the offset and relative orientation of trackingmarker 12 to define an origin of a coordinate system at fiducial key 10and to determine the three-dimensional orientation of fiducial key 10 inimage information, as shown in FIG. 4C, comprises the following steps inFIG. 11. The process starts with the controller, for example processor214 and memory 217 of computer 210 of FIG. 2, obtaining [at 4542] fromthe database geometric information about at least one pattern tag (forexample pattern tag 77) associated with the tracking marker 12, thecontroller determining [at 4544] within the image information thelocation of at least one of the pattern reference points of the at leastone pattern tag 77 based on the geometric information, and thecontroller determining [at 4546] within the image information therotational orientation of the at least one pattern tag (for examplepattern tag 77) based on the geometric information. With therelationship of the pattern reference point to tracking markerpre-established within the geometrical information, and the offset andrelative orientation of the tracking marker 12 with respect to fiducialkey 10 known (see step in FIG. 4C), a coordinate system is established[at 4548] at the fiducial key 10.

The rotationally asymmetrical tracking marker arrangements describedhere can be applied to other fields of general machine vision andproduct tracking beyond the field of surgery. More specifically, whiletracking marker 12 has been described in terms of being attached to afiducial key 10 by a tracking pole 11 (see for example FIG. 3B), thepatterned tracking markers of the present invention may be applied inother fields without the use of fiducials and tracking poles, in whichcase they are useful in determining the physical spatial orientation ofitems bearing the patterned tracking markers. By way of example, aflexible pattern tag may be applied to a cylindrical surface of anobject, such as a can in the food industry. With the pattern referencepoint known and with the mathematical description of the pattern known,the position of the can and the curvature of the pattern tag mayrespectively be determined from image information obtained using asuitable tracker.

In a further embodiment of the present invention, shown schematically inFIG. 12, a three-dimensional position and orientation tracking system,shown generally at 1200, comprises at least one pattern tag 1220attached to an item 1210, the pattern tag 1220 comprising a plurality ofcontrasting portions 1222. The system 1200 further comprises a tracker1280 configured for obtaining image information about the at least onepattern tag 1220; a database comprising geometric information describinga pattern 1224 on the at least one pattern tag 1220; and a controller1290, for example processor 214 and memory 217 of computer 210 of FIG.2. The controller 1290 is configured for receiving and processing theimage information from the tracker 1280; accessing the database toretrieve geometric information about the at least one pattern tag 1220;and comparing the image information with the geometric information. Theplurality of contrasting portions 1222 are arranged in a rotationallyasymmetric pattern 1224 and at least one of the plurality of contrastingportions 1222 has a perimeter 1226 comprising a mathematicallydescribable curved section. The perimeter 1226 of the at least onecontrasting portion 1222 may comprise a conic section including, forexample without limitation, an ellipse or a circle. The at least onepattern tag 1220 may be flexible. The at least one pattern tag 1220 maybe substantially planar.

In another embodiment of the present invention, shown schematically inFIG. 13, the three-dimensional position and orientation tracking system,shown generally at 1300, may comprise at least two pattern tags attachedto an item 1310, a first of the at least two pattern tags, shown in FIG.13 as pattern tag 1320, comprising a first plurality of contrastingportions 1322 and a second of the at least two pattern tags, shown inFIG. 13 as pattern tag 1330, comprising at least one contrasting portion1332; a tracker 1380 configured for obtaining image information aboutthe at least two pattern tags 1320 and 1330, a database comprisingpattern tag information describing the appearance of the at least twopattern tags; and a controller 1390, for example processor 214 andmemory 217 of computer 210 of FIG. 2. The controller 1390 is configuredfor receiving and processing the image information from the tracker1380; accessing the database to retrieve geometric information about atleast two pattern tags 1320 and 1330; and comparing the imageinformation with the geometric information. At least one of the firstand second pattern tags, taken to be 1330 in FIG. 13, has one or morecontrasting portions 1332 arranged in a rotationally symmetric pattern1334; the contrasting portions 1322 and 1332 of respectively the firstand second pattern tags 1320 and 1330 together constitute a rotationallyasymmetric pattern; and at least one contrasting portion 1322, 1332respectively of each of the at least two pattern tags 1320, 1330 has aperimeter 1326, 1336, comprising a mathematically describable curvedsection.

In respect of the two embodiments exemplified in FIGS. 12 and 13, simplepatterns have been used as examples. The patterns may be chosen to bemore complex and thereby more unique. This allows the pattern stages tobe uniquely identified within the field of view of the tracker. Thus avariety of items, objects, instruments or implements may be tagged withpattern tags, thereby to uniquely identify and track such items,objects, instruments or implements and determine their orientations.

In a further aspect of the invention, described at the hand of FIG. 14,a method is provided for tracking an item bearing at least one patterntag, for example pattern tag 1220, 1320, or 1330 of FIGS. 12 and 13. Themethod comprises a suitable controller 1290, 1390 (comprising forexample processor 214 and memory 217 of computer 210 of FIG. 2)obtaining [at 1410] from a suitable tracker, for example tracker 1280 ofFIG. 12 or tracker 1380 of FIG. 13, image information about the at leastone pattern tag. The method further comprises the controller 1290 or1390 obtaining [at 1420] from a suitable database geometric informationabout the at least one pattern tag (for example pattern tag 77), thecontroller identifying [at 1430] the at least one pattern tag on thebasis of its unique pattern, and the controller determining [at 1440]within the image information the location of at least one patternreference point of the at least one pattern tag based on the geometricinformation, the geometric information specifically comprising amathematical description of at least a section of the perimeter 1226,1326, 1336 of at least one contrasting portion 1222, 1322, 1332 of theat least one pattern tag. The method further comprises the controllerdetermining [at 1450] within the image information the rotationalorientation of the at least one pattern tag based on the geometricinformation. Having located the at least one pattern reference point andhaving determined the rotational orientation of the at least one patterntag, the user is queried [at 1460] as to whether the process shouldcontinue or not. If not, then the process is ended [at 1470]. If theprocess is to continue, then the process returns to obtaining refreshedimage information [at 1410].

The patterns 72, 82, 82′, 92, 92′, 102, 102′ of FIGS. 7-10 are all shownconfined within contrasting backgrounds that are rectangular. Inparticular, in FIGS. 7-10 the patterns 72, 82, 82′, 92, 92′, 102, 102′are shown as confined within contrasting backgrounds that are squares,which form a subset of rectangles. In yet other embodiments, thepatterns, as identified by their contrasting portions, may be confinedor configured within general non-square contrasting backgrounds. Weemploy the term “non-rectangularly confined contrasting portion” todescribe a contrasting portion of pattern not confined to a rectangularcontrasting background. Since a square is a rectangular shape, patternsnot confined to a rectangular contrasting background, are inherentlyalso not confined to a square background. By way of example, anon-rectangular contrasting background may be a parallelogram as in FIG.15, or triangular as in FIG. 16, or any other generalized shape.

In FIG. 15, pattern 152 on tracking marker 12 is confined by acontrasting background 157 having a parallelogram shape. Pattern 152 isrotationally asymmetrical. Parallelogram 157 may be the shape of thepattern tag employed, or may be a contrasting background on trackingmarker 12 itself, or may be a background on part of a pattern tag.Contrasting portion 154 has a contrasting portion reference point 158,which is the center of contrasting portion 154 in this exampleembodiment. Contrasting portion 154 has a perimeter 156 that isdescribable by a mathematical function, being a circle in this exampleembodiment.

In FIG. 16, pattern 162 on tracking marker 12 is confined by contrastingbackground 167 having a triangular shape. Pattern 162 is rotationallyasymmetrical. Triangle 167 may be the shape of the pattern tag employed,or may be a contrasting background on tracking marker 12 itself, or maybe a background on part of a pattern tag. Contrasting portion 164 hascontrasting portion reference point 168, which is the center ofcontrasting portion 164 in this example embodiment. Contrasting portion164 has perimeter 166 that is describable by a mathematical function,being a circle in this illustrative embodiment.

In yet further embodiments the patterns, as identified by theircontrasting portions, may be on backgrounds that have no particularconfining perimeter and may simply be on a contrasting background of ashape that is not predetermined. The background may be the surface ofthe tracking marker 12 or be part of the tagged item itself. We employthe term “unconfined contrasting portion” to describe a contrastingportion of a pattern having no predetermined confinement. FIG. 17provides an example of tracking marker 12 bearing, either directly onitself or on a pattern tag attached to it, unconfined pattern 172.Pattern 172 is rotationally asymmetrical. Contrasting portion 174 hascontrasting portion reference point 178, which is the center ofcontrasting portion 174 in this example embodiment. Contrasting portion174 has perimeter 176 that is describable by a mathematical function,being a circle in this example embodiment.

In FIGS. 3K, 3L, 7, 8, 9, 10, 15, 16 and 17 the patterns 313, 323, 72,82, 82′, 92, 92′, 102, 102′, 152, 162 and 172 are shown as comprisingdots, or, in a more general case, other shapes, distributed on gridpoints, the grid points themselves having a distinct predetermineddistribution, the dots being either present or absent at the gridpoints. In a further embodiment, the dots or other elements constitutingthe patterns may be randomly distributed, the random distribution beingincluded in the geometric information employed to identify and orientthe tracking marker. The use of random distributions of contrastingportions helps to ensure that the patterns on pattern tags are morespecifically unique and do not have rotational symmetry. Furthermore,the larger the number of contrasting portions, the more accurate thelocation and orientation of the tracking marker can be determined.Mathematical and other methods for generating random spatialdistributions are well known in the art and will not be further dweltupon here. FIG. 18 shows an example of a tracking marker 12 bearing,either directly on itself or on a pattern tag attached to it, unconfinedpattern 182 having a random distribution of contrasting portions 184.Contrasting portion 184 has contrasting portion reference point 188,which is the center of contrasting portion 184 in this exampleembodiment. Contrasting portion 184 has perimeter 186 that isdescribable by a mathematical function, being a circle in thisillustrative embodiment.

If the tracking marker, pattern tags and patterns of FIGS. 15-18 areemployed, then the database in question is pre-supplied with informationconcerning the tracking marker 12, the pattern tags 157, 167, 177, 187;the patterns 152, 162, 172, 182; and the contrasting portions, forexample contrasting portion 154, of the pattern tags. The informationcomprises, in particular, the mathematical descriptions of curvedsections of the perimeters of the contrasting portions of the patterntags, for example perimeter 156. It may also comprise the locations ofcontrasting portion reference points, for example contrasting portionreference point 158, and pattern reference points for pattern tags thatare be employed.

In embodiments in which the pattern on the pattern tag has a randomdistribution, the geometric information comprises predetermineddistribution information describing the random distribution of thecontrasting portions of the pattern on the pattern tag. In theseembodiments, the determining of the three-dimensional location of the atleast one pattern reference point and the determining of thethree-dimensional orientation of the tracking marker both comprisecomparing the image information from the tracker with the predetermineddistribution information within the geometric information.

Another embodiment of tracking marker 12 of FIGS. 3A and 3B is describedin more detail at the hand of FIG. 19. As stated heretofore, trackingmarker 12 may have a particular identifying pattern. The particularpattern, shown generally at 1972, comprises a plurality of contrastingportions 1974. As with pattern 72 of FIGS. 7-10, pattern 1972 is furthercharacterized by being rotationally asymmetrical. As a result, an imageof the pattern 1972 inherently identifies the rotational orientationabout an axis perpendicular to the plane of pattern 1972 of trackingmarker 12. Pattern 1972 is further characterized by having at least onecontrasting portion 1974 that has a perimeter having a mathematicallydescribable polygonal shape. In FIG. 19, the perimeter 1976 is shown ashaving a triangular shape in particular. In other embodiments themathematically describable geometric shape may be any polygonal shape,including but not limited to a square, a rectangle, a parallelogram, apentagon, a hexagon, or any other polygon. In a more generalimplementation, the geometric shape may be a distorted geometricalstructure in which one or more of the sides of the geometrical shape hasa different length from any other of the sides of the geometrical shape.

The pattern 1972 may be implemented on a separate pattern tag 1977 thatis attached or pasted, temporarily or permanently, to the trackingmarker 12. Conversely, the pattern tag 1977 may be in itself a trackingmarker, such as, for example tracking marker 12, so that the trackingmarker itself bears pattern 1972. The pattern tag 1977 may be planar.The pattern tag 1977 may be flexible to allow it to return to planarityafter being flexibly deformed. The materials of the pattern tag 1977 maybe, for example without limitation, a polymer or a paper or a mix ofboth paper and polymer. In other embodiments the tag 1977 may benon-flexibly deformable while remaining dimensionally stable. Anindividual tracking marker may comprise a plurality of pattern tags,each with a pattern of its own, as will be described below.

In this embodiment, as with those described at the hand of FIGS. 7-18and the associated method, the database in question is pre-supplied withinformation concerning the tracking marker 12, any pattern tags ontracking marker 12, the patterns (for example pattern 1972); and thecontrasting portions, for example contrasting portion 1974, of thepattern tags. The information comprises, in particular, mathematicaldescriptions of the perimeters of the contrasting portions of thepattern tags, for example perimeter 1976. It may also comprise thelocations of contrasting portion reference points, for examplecontrasting portion reference point 1978, and pattern reference pointsfor pattern tags that are be employed. In FIG. 19, reference point 1978is taken to be the centre of contrasting portion 1974 bounded bytriangle 1976 as perimeter.

In terms of structure, method, use, and operation, all the conceptsdescribed at the hand of FIGS. 7-18 also apply to this embodiment, withthe only difference being that the perimeters of the contrastingportions are mathematically describable polygonal shapes.

While this invention has been described as having an exemplary design,the present invention may be further modified within the spirit andscope of this disclosure. This application is therefore intended tocover any variations, uses, or adaptations of the invention using itsgeneral principles. Further, this application is intended to cover suchdepartures from the present disclosure as come within known or customarypractice in the art to which this invention pertains.

What is claimed is:
 1. A three-dimensional position and orientation tracking system comprising: a tracking marker bearing at least one pattern tag comprising a plurality of contrasting portions, at least one of the plurality of contrasting portions having a polygonal shape, the tracking marker having a shape with a surface that extends in three dimensions, a tracker configured for obtaining image information about the at least one pattern tag; a database comprising geometric information, the geometric information containing a pattern on the at least one pattern tag and a mathematical description of the polygonal shape of the contrasting portions; and a controller having a processor and memory, the controller in communication with the tracker and the database, the memory storing software that when executed by the processor is configured to receive and process the image information from the tracker; access the database to retrieve the geometric information; and compare the image information with the geometric information to identify and obtain a three-dimensional orientation and a three-dimensional location of the at least one pattern tag.
 2. The system of claim 1, further including a display in communication with the controller, the controller further including software that when executed by the processor displays one of the position of the at least one pattern tag in the image information from the tracker, potential trajectories of the pattern tag, and boundary conditions relating to the at least one pattern tag.
 3. The system of claim 1, wherein the controller further includes software that when executed by the processor creates a model based on a previous scan and image information from the tracker, said system further comprising a display in communication with the controller and the controller includes further software that when executed by the controller displays the position of the at least one pattern tag within the model on the display.
 4. The system of claim 1, wherein the perimeter of the at least one contrasting portion is a square.
 5. The system of claim 1, wherein the perimeter of the at least one contrasting portion is a triangle.
 6. The system of claim 1, wherein the perimeter of the at least one contrasting portion is a hexagon.
 7. The system of claim 1, wherein the at least one pattern tag comprises a flexible material.
 8. The system of claim 1, wherein the three-dimensional surface is at least a segment of a cylindrical surface.
 9. The system of claim 8, wherein the segment is a ring.
 10. The system of claim 1, wherein the three-dimensional surface is an ellipsoid surface.
 11. The system of claim 10, wherein the ellipsoid surface is at least a segment of a spherical surface.
 12. The system of claim 1, wherein the tracker is a non-stereo tracker.
 13. A method for tracking an item bearing at least one pattern tag, the pattern tag bearing a rotationally asymmetric pattern having a plurality of contrasting portions, at least one of the plurality of contrasting portions having a perimeter with a polygonal shape, the method comprising the steps of: obtaining from a tracker image information about the at least one pattern tag; obtaining from a database geometric information about the at least one pattern tag, the geometric information comprising a mathematical description of a polygonal perimeter of the at least one contrasting portion of the at least one pattern tag; determining within the image information a three-dimensional location of at least one pattern reference point of the at least one pattern tag based on the mathematical description of the polygonal perimeter, and determining within the image information a three-dimensional rotational orientation of the at least one pattern tag based on the geometric information.
 14. The method of claim 13, wherein the plurality of contrasting portions has a predetermined random distribution; the geometric information includes predetermined distribution information about the random distribution; and the determining the three-dimensional location of the at least one pattern reference point comprises comparing the image information with the distribution information.
 15. The method of claim 16, wherein the step of obtaining from a tracker image information involves obtaining non-stereo image information.
 16. A surgical monitoring system comprising a tracker for obtaining image information of a surgical site; a fiducial reference configured for removably attaching to a location proximate the surgical site; a tracking marker in fixed three-dimensional spatial relationship with the fiducial reference and observable by the tracker, the tracking marker bearing at least one pattern comprising a plurality of contrasting portions and at least one of the contrasting portions having a perimeter with a polygonal shape, the tracking marker having a shape with a surface extending in three dimensions; and a controller configured to spatially relate image information to previously obtained scan data, the controller having a processor and a memory, the controller in communication with the tracker, the memory storing software that when executed by the processor determines the three-dimensional location and orientation of the fiducial reference by relating the image information to the scan data on the basis of the polygonal shape.
 17. The system of claim 16, further including a display in communication with the controller, the controller further including software that when executed by the processor displays one of the position of the pattern tag in the image information from the tracker, potential trajectories of the pattern tag, and boundary conditions relating to the fiducial reference.
 18. The system of claim 16, wherein the controller further includes software that when executed by the processor creates a model based on a previous scan and image information from the tracker, said system further comprising a display in communication with the controller and the controller includes further software that when executed by the controller displays the position of the pattern tag within the model on the display.
 19. The system of claim 16, wherein the perimeter of the at least one contrasting portion is a square.
 20. The system of claim 16, wherein the perimeter of the at least one contrasting portion is a triangle.
 21. The system of claim 16, wherein the perimeter of the at least one contrasting portion is a hexagon.
 22. The system of claim 16, wherein the three-dimensional surface is at least a segment of a cylindrical surface.
 23. The system of claim 22, wherein the segment is ring.
 24. The system of claim 22, wherein the three-dimensional surface is an ellipsoid surface.
 25. The system of claim 24, wherein the ellipsoid surface is a segment of a spherical surface.
 26. The system of claim 16, wherein the wherein the tracker is a non-stereo tracker.
 27. A method for tracking an item bearing a pattern tag having at least one unique rotationally asymmetric pattern, the method comprising the steps of: obtaining image information from a tracker about the at least one pattern tag; identifying the at least one pattern tag within the image information on the basis of its unique pattern; obtaining from a database geometric information about the at least one pattern tag, the geometric information comprising a mathematical description of a polygonal perimeter of at least one contrasting portion of the at least one pattern tag; determining from the image information the location of at least one pattern reference point of the at least one pattern tag based on the mathematical description of the polygonal perimeter; and determining from the image information the three-dimensional rotational orientation of the at least one pattern tag based on the geometric information and the at least one pattern reference point.
 28. The method of claim 27, wherein the step of obtaining image information from the tracker involves obtaining non-stereo image information from the tracker.
 29. A method for tracking the location and rotational orientation of an item comprising the steps of: attaching to the item a tracking marker bearing a rotationally asymmetric pattern of contrasting portions, at least a section of the perimeter of at least one contrasting portion of the at least one pattern tag having a mathematical description; obtaining image information from a tracker about the at least one pattern tag; obtaining geometric information from a database about the at least one pattern tag, the geometric information comprising a mathematical description of a polygonal perimeter of at the least one contrasting portion of the at least one pattern tag; determining from the image information the three-dimensional location of at least one pattern reference point of the at least one pattern tag based on the geometric information; and determining within the image information the three-dimensional rotational orientation of the at least one pattern tag based on the mathematical description of the polygonal perimeter.
 30. The method of claim 29, wherein the step of obtaining image information from a tracker involves obtaining non-stereo image information from the tracker.
 31. The method of claim 29, wherein the plurality of contrasting portions have a predetermined random distribution; the geometric information includes predetermined distribution information about the random distribution; and the determining the three-dimensional location of the at least one pattern reference point comprises comparing the image information with the distribution information. 